Et 159 223 

lUTHOE 
TlTtE 



POB DATE 
BOTE 



EDBS PBICE 
DESCBIPTOES 



IDENTIFIERS 



DOCOHEIT BESOHB 



, TH 007 706 



Olejnllc, Steplien P,; Porteir Andrew C, , 
Efitiaating Treatment Effects and Precision for 
Quasl-Experi^ents Acisuning Differential Group and 
Individual Growth' Patterns, r . 

[Mar 78] - C 

18p,; Paper. presented at the Annual Meeting, of th^ 
Anerlcan* Edicational fiesearch Associa:tion (62nd, 
Toronto^ Ontario, Canada, ^Harch 27-31, 1978) 

MF-$0.83 HC-$1.67 Piriis Postage. 

^Acad^mic Achieveaontj ♦AchievejBent Gains; ♦Airalysis 
of Covariance; Analysis of Tariance; Conparatlve 
Statistics; Hypothesis Testing; "i^Mathenatical^Hodels; 
^Beliability ; t'l^esearch Design; Sanpling; Standard 
Error of Measurement; Tirfst Heliability; Tests of 
Significance 

Fan Spread Hypothesis; t'Quasi Experimental Research; 
♦Sample Size 



ABSTRACT 

The statistical properties of two methods of 
estimating gain scores for groins in guasi-e^periments are compared: 
(1) gains in scores standardized separately for each gi:oup; and (2) 
analysis of covariance with estimated true pretest scpres. 'The fan 
spread bypothesis^is^ assumed for groupis but not necessarily assumed 
for members of the groups.^ Sample standard fd^eviatxons provide a 
biased estimate cf treatment ef^fects when the first , procedure is»used 
with small samples and poorly correlated measures. Oiider a nonlinear 
model of within-group growth, the true scoife analysis of covariance 
underad justs for in^-tial group differences , Under the. assumption of 
linear -within-group growth and with large samples, both methods 
estimate the desined' .effect with eguali precision. When samples are 
small, neither of the two methods are ^approp relate for the nonlinear 
'model of within-group growth. (Aiithor/CTM) 
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Abstract 

The appropriateness of standardized gain scores and analysis 
of covariance adjusted for errors of measurement wej^MB , considered 
for quasi-experiments conforming to the fan spread hypothesis. 
Previous confusion in this ;'girea was resolvetii by considering a 
linear and a non-linear model of within group growth. For ;the 
linear model both procedures estimated the 4esired effect with 
equal precision when samples were large. With small samples analysj^ 
of covariance was appropriate but. standardized gain scores" estimated 
the wrong effect with spurious' power. For the non-linear model only 
standardized gain scores were appropriate with large samples. 
Neither procedure was appropriate for the non-linear model with 
small samples. 





'',1. 



V Selecting an appropriate ^okiysia strategy i*dr a s^^tudy Jb^ajed 
oni a <iuasi-e;?perimental resiearch design haS' been ii, topio ot •on- 
slderable controversy. The noneqi^ivalent coritrbl g3b5>\>^ Resign 
(Gftmpbell and Stanley, I963) in particular has received a \grea,t deal 
^ attention. Recently, the discussion has focused. on thp\issue of. 
alcademic rate of growth and its impl'icfationa for traditioniawk^an&lyses 
procedures. Specifying the appropriate analytic model is dependent 
on how individuals change over ti^ne. Some authorities (Campbell and 
Borucht 1975) have suggested that the! initial 'difference between the 

■ ■ * . •■ ' ' ( i"''' . ■ •: ■ ' 

comparison groups dti a pretest achievement measure implies that the ^ 
comparison groups are growing academically ^t different rates. ^ 
Initial achievement' differences havQ been found frequently in qiiasi- ■ 
experimental ^studies like the evaluation of compensatory e^jiuc^ 
programs. This -differential growth rate probl'em has beerl labeled the 
fan spread hypothesis (Campbell and Erlebacher, 1970).' The theory * 
suggests that wi^Jhoiit a treatment, the differ^nc'el between the 
comparison group means would increase over time 4nd there would be ^ 
a proportional increase in the' within group variab^ity. Th|.y . ,/ 
relationship between the increasir?tg' me>n difference\nd. th 
group variance can be. represented algfebraically as the YollQwingi 

• u ' ' ■ ' ' * : ^' * 



wher,e: . ^ . 



u 4. are the population means on measure (X) for the 

^'xpf ^xct p^ogr^m and control groups, respectively, at time t; 



o 



is the pSWed wi thin-group standard deviation .of the ^ • ^ 
* outcome measUr*e at- time t; StndV 

is a constant • ; \ > • \. y 



The use of tr^c^itional analyses techniques under this concept- 
ualisation ha^beeit challenge^ as^ inappropriate on the ba^is that 
they underad^us^ fbr groujp differences (Campbell and Boruc^ 1975i 
Kenny » 19761. I^ot everyone agrees however with this assessment 
fi^rid evidence sui)porting the use of tradi^tional analyses procedures 
have been presented , (Porter, A. G. and ChibucoSr T., 197^1 
Bi^yk, A.S. and W^isberg, H. . 1977). 

The confusion corytirerning this issue of an appropriate 
analytic strategy Canlbe attrl^buted to a large extent to conflicting 
asBump.tioJis made implicitly on' the nature of individual growth. 
The previous discussions of the fan spread hypo t he s'is have 
concentrated on' the differential growth rates between comparison » 
groups.. The growth rates for these groups have been assumed to 
bje linear over time. The growth rates of individuals within these • 
groups however have been ignored. For the average group growth 
to be linear, individuals may grow linearly or non-lihearly . , ^ 

A linear model of within group growth implies that individuals 
are-^ changing at a constant rate (see figure 1). The correlation 



er|c 




c 




gure i. 



The fan spread hypothesis with the linear model 
of within group growth.. The solid lines represent 
average ^r^up growth while the dashed lines re-. ^ 
pre^sent inaividual growth. 



ERIC 



between a ituretest and 



Jia pdSttest* should therefote equal unity 
' ejfcept for inea8U][:emen| errors. The nonP^inear model of^witJiiri. 
gr^up.gcbv^^th suggests|that an individual grow academical]^ 
at varyir|^g*tates acro|^ is-, academic growth may 

0ccur in* spurts (see |[*igurf ' 2); The correlation between a pretest 




y ■ Figure .2 #f Thp|faYi spread hypothesis with a n'ort-linear' model 
/ • ^ of within group growth* * ^ ' ■ - 

and a poSttest measjire under; this %r\odel wp.uid not equal unity. . 



even with a perfectly relia^ble iristrument,^ Selecting the appropriate 
analysis strategy d|pends on whiph of these .models of within : 



group, growth is^ appropriaj^a 

■ ■ ' " , " . r, ' . 

Purpose L 



The purpose oif the jpres^ent study, was t6 consider 'two competing 



' analytic str^tegiei^^ in li^ht bf ^ of within^group 

growth suggested al|bVe, ^ The twc^ procedures coas^idgred were the 
^olloWingi* 1) stfemdardi^ed gain soothes or more approjiriately gains 



in standard scoresJarid 2) ?ingl^ , covaritible analysis of covarlance, 

- ■ - ■■■■ •'■ 7 ■ ■ ■ • ' 

wljth estimsite^ tnu^ scores. > Th the 

■ f< - • ■ ■ ■ '/-"i-^'''" ■ - ■ ■ ". ■ .- 

,usL of the analysi^ c^, variance model with the dependent variable 

"created by taking fthe difference be tweeh a' pretest a;n(i' a poaAitest 



6> 
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a^Wr standardizing each measure. Standar<Jl-^tion is achieved by . - 

■ ■ ' '. ■ °. ■ ' ■'■.■■•,« ' - • ■■■ ' .. .■ 

.dividi/fig'each measuRe by the pdoled within group standard deviation ♦ 

I . \ • ■ • )'•'■■■.'• ■. . ■' . . . . 

at eadh point in time. The secolid technitiue uses the estimated 

true" \sb pre of the pretest rather than the obs^erved pretest sco/f-e • 

afe^the •covariate||||^ the analysis of covariance model (Porter, 1967) - 

r- Thia procedur^^^o'rrects for the errors of measurement found ' 

in' the pretest data. . The appropriateness of each technique under 

the twd .'models of wi'thip group growih was based on two criteria* 

<1) the effect es'timated by each procedure and 2) the precision 

with which each^effect was estimated. The two analyses procedures 

studied hav^ been considered -preyiously in the literature .on the- 

^aWs of the first criterion (Kenny, D.> 1975% Bryk and Weisb?rg^t 1977)* 

Becki/se the na-ture of within group growth assymed was not Explicitly 

ststied, 'the results and recommendations have been contradictory. 

The/second criterion proposed m the present study has not beeh 

Jnsidered in the previous studies examining the two competing 

^ • , ^ ' . , ' <. 

analytic sti^ategies. Precision provides a basis on which an 

anaiysis technique may be selected in situation's where the saine - 

desired effect is-^^stimatod by two or more procedures. 



Estimation under the fan spread hvpothesis 



The fa9^ spread model of growth suggests that concomitant with 
an. increase in mean- difference between comparison groups is "a . \ 
proportional iiricrease in within group variabij.ity. Furthennore 
this relationshfp<be"tween the m^an difference and pooled standard 
-deviation remains constant aaross time. Algebraically this 
relationship can be presented as the^f^tlbwingi <> . 



ERLC 



The terms arei^dls d«(fined previously. From 1;his definition of 
the fan spread Kypothesis, the appropriate adjustment for 'differences 
on a posttest (Y)t 'given no treatment effedtt is the. product of^ . 
th^: pretest difference and- the ratio of posttest to pretest 
stahdeird deviations i / \ ^ ^ 

Ah anaJLj^tic strategy having .this ad justment factor ^^provides ar^ 
unbiasec^estimate- of group differences in situations conforming 
to the fan spread model of growth. SiVlce th^ definition of the ^ ► 
fan spread hypothesis doeSHftoi Include a re.ference to' the nature 
bf 'within group growth patterns,^ the' above ad^justment is - 
appropriate for both the linear and non-linear niod61s of within 

group growth* , * 

J. ' ' ■ _ 

Eatlmati n-n with fjrains in standard scores . • , 

* Standardizing the scores at each point i/i time adjusts for 

the increasing wt^thin group variability suggested by the fan , 

spread ^odel of growth./The effect estimated by computing the ^ - 

uandard scores can be written as the followingi ^ \ 



/ 'I 
gain^ m s^ 



I 



: ^Gss V- V'o^ Kp-^t^• 



This estimate is identical to the adjustment strate^ suggested 
above based, on the definition of the fan spread model of growth. 
Thus' gains^ in standard scores provides an unbiased^^edtimate bf * 
a /treatment effei^t in situations conforming -to the fan spread 
model. A' problem arises however when standardization is achieved 
by using\the pooled within groups standard- deviation calculated 
on the sample data.' Tha L is, the expected value of the ratio- 
of) pooled withia group standard deviations, ^/^jc do^s not equall 



the desired ratio of the population standard deviations. Table 1. 
presents 'th^ expected value of the ratio of two non-independent 



Table 1 



V 



^The expected value of the ratio of two non-independent' 
f^tandard. deviations assuming equal populfitign st^riiJatrd 
deviations. „ "^"^^^ 



n 


e=.9 


^-.7 




• 20. 


1.0055' 


1.0146 


^.0238 




1.0026 


l.(ip68 


■ 1.0112 


.60 


^ 1.001? 


1.0044 


1.0073 


80 


1.0012 


* 

1.0033 


•1.0054 


100 


1.0010 


1.0026 ^ 


1.0043 




sample Standard deviations for various sample sizes and ^elected 
rela;cionqhips between the two measures when the population 
starfHard deviations are equal. If the population standaf'd 
deviations are unequal, as suggested , by the- 'fan spread hypothesis, 
the descrepencies- are even greater. Thus the use of the sample^ 

■» ■ 

standard deviations to standardize scores provides a biased 
estimate o*f a treatneat effeo^ in studifes involving small sanples 
and poorly correlated measures, large samples* arid highly 

correlated measures however the procedure is appropriate. 
Furthermore, since the' technique is not affected by the relation- 
ship among the/ individuals within comparison groups, ^the gains 
/ in standard scores is appropriate for both the lir^ear and non- 
linear models of individual growth., * 
Estimation with^true sco re an^ilvsis of covarianc'e 



The second ahalytic strategy^ considered "for situations conforming 



' ' ■ ' ^ - ■ ■ ^ r ' - ' 

tb the 'fan spread model was the true score ^acnalysis of covarianae , 

proceduce, * The proceclure assUmes -knowledge of the ap)propriat.e ^."^ 

population reliability cpefflcient in order to calculate the \ ' 

'true scorigs of the pre tesj^ data. The effect estimatefi by using 

the true scores as a covari&te in 'the analysis jpf covariance model^ 

can be writt^en as the following! , ' 

.Where -^^^ and j^^^ ^re the population correlation coefficient and » 
reliability qoefficient; respectively. The other tei^.s are as 
defined previousty. The abova estimate is identical to , the - v 
adjustment strategy suggested by the fan spread definition except 
for the ijatib of ^*V<xx* "^^^^ ratio provides th?^ correction for 
errors of^easurement. Thei^e^f if the true relationship between 
the two measure^^ is pe^rfect, proposed by the line'kr model of 
within group growth, the ratio of tW correlation to the 
reliability of the covariate will also equal .unity. » Thus for the 
linear model of . within group growth the analysis of (^variance 
model\ith estimated true scores provides an appropriate adjust- 
ment for the fan spread, situation,' 

Under the non-linear model of within grbup growth, the true 
relatlons^iip between the' pretest and po&ttest' does not equal unity, 
even afte^r co^rrecting for errors of ' measurepient. Given this ^..^ 
model within group growth/ true score ^nalysis of coyariance 
underadjusts foi^ the initial group di^erences. • 

precision . - * ^ / . V' 

The above discussion on estimation showed, that both strategics 
Estimate the desired effect for the linear mod«l of withi £;roup jt: 



growth;/ In this situation^ the precision with, whicrh eacW, technique 
es^^nates the effect is fif great interest. It provided a ^s is / 
ori which a researcher coiri^d^select b6twedn the corfipeting pro- 
cetiures. Precision is define^l^in terms of the variability of the 
ef feist estimated, The greater/the variance of the. estimate the 
iQ^ar €he precision of the test. An indq^p, of pr-ecision is 
therefore found in the standard error of the simi)le contraefVof 
interest. 

Both' gains ijn standard scores ^d true ^score analysis of; 

covariancei^Van be conceptualized as an i-ndex of -response having the 

following formi ' ' % / 

„ ^ W Y - KX , 

Where I ^ \ ^ : \ 

',W 'is the adjusted variable; . 



X»y ar^e the pretest and postte^t pea sure s respectively; 
and. K is the adjustment coefficient, ^' ijl'" i 

The^^^tWo^ procedures differ in how the adjustment coefficient j.s . 
determined. The gains in standard sc^jres approach defines 'the 
.adj]ustment coefficient as the raH;io of the pooled within group 
standard deviations of posttest to pretest, -Sy /S^ , The 
adjustment coefficient for the' true score anaJLysis of covariance 
on the other hand is defined as the ratio of the. poole^ within 
group^^egression slope of postt^est on pretest to the population 



test-i^etest reliability coefficient^j^jG^^ model, tlie 

contrast of interest is the simple difference between the group 
means on tjie adjusted variable W,^ The indexf of preoisLon by V. 
whiciJ the two' strategies -e§n^ be compared is the following i 



The variance 6t the contrast can written as the^f pllowingi . ' 

ir{^^-%) « 2 tvar(J)+.(E(k')Wr(K))/arU^ 

I e 4^(>U^^ .U,^ )Var(K) ' / ^ 

(Kee sling and WiI6y, 1977) ' - . ^ V ^ . 

In baiculating^^the adj^asted vatiable' both gai/xs^ in standard ecor^s^ 

and true score analysis of covariance ; require ;1fha.t the ad justment 

coeKAqient ' cut) be estimated on the sample, Th^^s estimate Is 

likely to change from Sample- to sampl'e, ^'The adj-ustment coe:^;^|||^ient 

is' therefore a rdtndom variable which must be takian into /consideration 

by includinjg the 'Var(K) as f^ac^or in the.^ variance of ' the CQrttrast, . 

Since both strategies being coiisidered have,^ the * same f onn or ^'Ihe 

variance" of the effect esticnated, differences in precision must 

reauli; from dif:^6rences in the expected value and variance of 

the adjustment coefficients. " \ / ^ 

■ f - /: ' . ■ ■■ " ■ ^. • '-^ ■ -■ 

' Ihe expec+^d values for the ratio of two non-^independent 
standard deviation^ were presented in- table !• The adjuptment ^ • 
coef f icient^.f or true score analysis of covariance assumes that * 
the pppulationtest-retest reliability coeffiicient^ ip, known* Tha 
pxpec ted \^alu;e for this adjustment factor is 'there'for9^/e<^4> to 
\ ^ E(b^,jj ) • Since -^the . Sample regression slope is" an unbiased 
estimator of the population regression i^lope the following is 

/ ■ 7 \. \ ;. ... •■ ^ ^ ■ r ' ' ■ 

true i E(b^.^ ) = -Ccy ^ • For the lineslr model of within group growth 

the expected value of the true score adjustment coefficient- 

^ ' ... . ■ . - , ^ 

\ therefore equals unity since "^{Vx equal pretest and 

posttest variance have been assumed. ' f 

The variance of tfte adjustment co'p^iTi^ient for ' the gains' in * 
standard ^core-s procedure was derived frpm the density function . 

^for the ratio of "two non-independent standard deviations (o:]^ejhik, 1977) 
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Table 2 presents the variance for the adjustment coefficient 
given varying sample sizes anri selected relationships between the 

^ * '■ 

Table 2 

The variance of the ratio of two non-independent standard 
deviations assuming , equal population standard deviations. 



n 


^-9 


^..7 


^,-.4 


20 


.01X25 


.03049 ■ 


.0506.6 




.00515 


.01388 


.02296 


60 


.0033^* 


.00899 


.01485 


ao . 


.00246 


.00663 


.01096 


100 


.00196 


•.00527 


.00870 



"^two measure^v Th§ variance of the, true score adjustment coefficient 
.assuming knov^ledge of the. population test-re test reliability 
coefficient equals 1 Var (b^.^^ ) . The variance of the sample 
regression slope is equal to -^^-^ ^ . Table 3 presents for 
varying- sample sizes and selected relationships between the two 
measures, the variance of this adjustment coefficient. As with 
the previous tables the variance of the pretest and posttest have 



^ TSble 3 



\The.variance of the adjustment coefficient suggested by the 
true 5core analysis of covariance assuming equal pretest ^ 
and posttest variance 



n 


■e--9 






20 


.01379 


.06122 


.30883 


40 


.00634 


.02813 


.14189 


60 


.00412 


.01826 


.09211 


80 


.00305 


.01360 


.06818 


100 


.00242 


.01073' 


.05412 



been assumed to be equal. Finally to facilitate the comparisoi^ 

between the two procedures in terms of precision, table 4 

... • . . • .- y • • 

compares the sum of the expected value sq.uqjr.ed and the variance 

of the adjustment coefficient for both strategies. ' . 

; Table k ' - 

The sunv, of the expected value squared and variance (K ) +Var (K )J , 
for the, gain in standard • scores and true score analysis of 
covariance assuming equal pretest and posttest. variance 
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Gains in 
Standard 
Scores 



True Score Gains in 
Analysis of Standard 
Covariance Scores 



True Score Gains in 
Analysis of . ^Standard 
Covariance Scores^ 



True Score 
Analysis of^ 
Cova^piance 



1.01380 
1.0063i+ 
l.OOJ^ll 

1.00305 

1. 002^+2 



"1.05^89 
1.02753 
1.01781 
1.01321+ 



K= >^ 
1.0612^ 



i. 02813 

1.01826 
1.01352 
1.0ip73j 



1,09883 
1.0^+5^9 
1.029^9 
1*02179 
1.01732 



i'. 3cr&83. 

1. 1^+189 
1.09211 
1.06818 
1'. 05^+12 



1.02228 
1.01036 
1.0067^ 

1.00i^86 

1, 00396 

Digcussion 

A comparison , of the expected value , and variance associated 
with the adjustment coefficients for the two competing analytic 
strategies indicates almost no differences when the relationship 
between the pretest and posttest is high, .9 or .?. 

This is true regardless of the sample size. When the relationship 
between the measures is low, some, differences become 

apparent. In situations where the relationship is low but the 
sample size is large the two procedures appear to have equal 
precision. As the siample size decreases, greater precisibn is 
associated with the gains in standard scores approach. . 



12 . - - • ^ . , 

s ■ ♦ 

/- . ' 

The alcove conclusions on precision have beer; based on the , 
standard error presented in this pap^r. * There is, however, an 
important difference between the calculations provided here an4 • 
the standard error generally associated with the gains in. . . 

standard scores procedure. In actual practice ' gains in Btandard 

w y 

scorejs are deternuned in a two-stage process r first the adjusted , 

V . . . • 

v?Lriable is determined; and second it is^sed as tha dependent 

variable in the analysis of variance model. This pj^ocedure 
assumes that the. adjustme'nt coefficient deter;nined in step one 
is a constant^adross replications. This standard error therefore- 
differs from the correct standard error presented earlielr by 
eliminating all factors involvings the variability of the ad- 
justment coefficient. This reduced form of the: sfcxirxdard error ^ 
produces spurious precision which lead^ to a liberal test of 
the hypothesis under investigation. ? 

The degree to which the gains in standard jscores procedure 
as generally /determined, is too liberal, is dependent on the 
variability of the adjustment coefficient. Table 2 provides , 
some information on this question. When the relationship between 
the pretest measure and the posttest measure i^s high and the 
sample j^j^ze is large, the variability of ' the adjustment coefficient , 
is essentially zero. As discussed earlier these are the only 
conditions under which the procedure .estimates the porrect 
effect. ' Thus under those conditions the gains in standard scores 
procedure, as they are generally calculated, is likely to be appropriate. 
As the\8ample becomes small and the relationship weakens, the prob- 
ability of error increases as does the bias in estimating the effect.. 

\ -1 C 



These observation^ were based or.- the situation when the variance 
of the measures are equal across^^ time. In situations conforming 
to^ the -fan spr.i-ac r.orel ,• the' variability associated wi^th the 
adjustment coefficient increases and with it the inappropriat'eness 
of the gains in standard, scores techniqup. Thus, unless the ^ 
sample is large and the relationship between measures^is high 
the use of gaind in standard scores as generally calculated 
should be avoided. On the other hand, the estimated true score^ , 
analyb^is of covariance technique, .does take the variability of 
the adjustment coefficient into, consideration in determining its 
standard error. The analysis of covariance is , not a two -stage 
process but rather the adjustment coefficient and the estimation 
:;are computed in a single analysis. The disc.ussion presented 
here, has assumed that the population testAretest reliability 
coefficient was known. If it^must be estimated on the data set 
or an independent sample, the variability of the reliability 
coefficient must be,' taken into consideration in calculating the 
appropriate standard error, /Substituting a sample estimate of 
the test-retest reliability coef f icient . in the true score 
analysis of covariance approach can ^Iso result in spuriously 
high precision. Given that the population reliability coef f icient 
is knpwn, the. true score analysis of covariance provides an^ 
apprppriate estimate of ± treatment effect under t^e linear 
moder of within group growth and greatir' percision th^n the 
gains in stanriard scores technique. , 

The present study has resolved the problem* of conflicting 



recommend-itioris found in- the literature concerning the gains in 
stand ird scores and" analysis of covari^nce adjusted lor errt 



of measurement as .appropiriate analysis strategies in quasl- 
experiments conforming to the fs^n spread hypothesis. fhe 
confusion resorted from' conflicting ifnplicit assumptions on the , 
nature of incj'iyidual gfowth patterns. By c;Learly stating the ' 
nature of, within • grj-oup academic growth, th : appropriateness o^ 
'the two compe.ting analytic strategies were evaluated.^ ^"^^ 
linear model- of^ifiihln feoup growth with largB samples botl^Jgains 
in -standard scores and true score analysiS'of covariance es-timate 
the desired effect with approximately eqtial prte^ision. With small 
samples howe/er, the true score analysis of covai!',iance technique 
'estimates the desired effect with the appropr'iate'; standard^ error > 
but the effect"^ estimated by +Ke chains in standard scores approach 
is noli -the desired one and the estimate is made with spurious 
power. For the non-linear model of within group growth true 
score analysis of covariance does not estimate the desired effect 
Gains in standard scores dofe,s estimate the desired effect with 
the appropriate standard error only when the samples jare large. 
When samples are small neither of the two strategie^s considered 
are appropriate for the non-Uneai" model of wi'=bhin group growth. 
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